Skip to content

Android SDK build scripts #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open

Conversation

marcprux
Copy link

Following up on swiftlang/github-workflows#106 and swiftlang/swift#80788, this PR contains a new swift-docker/swift-ci/sdks/android/ folder with scripts that will build and upload a full stand-alone Swift Android Swift SDK. It is modeled after the structure of the swift-ci/sdks/static-linux build scripts.

Running

The top-level ./build script installs a host toolchain and the Android NDK, and then invokes scripts/fetch-source.sh which will fetch tagged sources for libxml2, curl, boringssl, and swift.

It then applies some patches and invokes scripts/build.sh, which will build the sources for each of the specified architectures. Finally, it combines the NDK and the newly built SDKs into a single artifactbundle.

Specifying Architectures

By default all the supported Android architectures (aarch64, x86_64, aarmv7) will be built, but this can be reduced in order to speed up the build. This can be useful, e.g., as part of a CI that validates a pull request, as building a single architecture takes around 30 minutes on a standard ubuntu-24.04 GitHub runner, whereas building for all the architectures takes over an hour.

To build an artifactbundle for just the x86_64 architecture, run:

TARGET_ARCHS=x86_64 ./build

Installing and validating the SDK

The .github/workflows/pull_request.yml workflow will create and upload an installable SDK named something like: swift-6.1-RELEASE_android-0.1.artifactbundle.tar.gz. The results of one of the workflow runs can be seen at https://github.com/swift-android-sdk/swift-docker/actions/runs/14603885089

The workflow will also install the SDK locally and use swift-android-action to build and test various Swift packages in an Android emulator.

marcprux and others added 5 commits March 31, 2025 14:28
* Build Android image

* Checkout without ssh

* Retry build if it fails

* Swift 6.1 Release Dockerfiles (swiftlang#456)

* Change binutils-gold package dependency on Debian 12 to binutils (swiftlang#457)

* Update installed packages after nightly platform expansion (swiftlang#458)

* update nightly-6.1 dependencies

* update nightly-main dependencies

* fix ubuntu images

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Fedora 41 Dockerfile (swiftlang#464)

* Build Android image

* Build Android image

* Swift 6.1 Release Dockerfiles (swiftlang#456)

* Change binutils-gold package dependency on Debian 12 to binutils (swiftlang#457)

* Build Android image

* Build Android image

* Build Android image

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Android SDK build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Androd build

* Swift Android build

* Swift Android build

* Swift Android build

* Swift Android build

---------

Co-authored-by: Mishal Shah <shahmishal@users.noreply.github.com>
Co-authored-by: Chris McGee <87777443+cmcgee1024@users.noreply.github.com>
Co-authored-by: Justice Adams <107649528+justice-adams-apple@users.noreply.github.com>
Co-authored-by: Andrew Sukach <134116196+sookach@users.noreply.github.com>
* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Start splitting NDK out from the rest of the SDK

* Start splitting NDK out from the rest of the SDK

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2

* Swift Android build 6.2
* Swift Android build 6.2

* Swift Android build 6.2

* Add post-install script to SDK bundle

* Add post-install script to SDK bundle

* Add post-install script to SDK bundle

* Add post-install script to SDK bundle
@marcprux
Copy link
Author

marcprux commented May 8, 2025

I've updated this PR to build the SDK without including the Android NDK. In order to accommodate this, there is now a post-install scripts/setup-android-sdk.sh that will need to be run manually after the swift sdk install … command. This will find the NDK based on the standard ANDROID_NDK_HOME environment variable and create links from the NDK into the path specified by swift-sdk.json's sdkRootPath, as well as pulling in the swiftrt.o to work around swiftlang/swift#79621.

The latest result of the build for the 6.2 nightly can be seen at: https://github.com/skiptools/swift-android-toolchain/releases/tag/6.2-DEVELOPMENT-SNAPSHOT-2025-05-07-a

@marcprux marcprux marked this pull request as ready for review May 8, 2025 19:30
@marcprux marcprux requested a review from shahmishal as a code owner May 8, 2025 19:30
@marcprux
Copy link
Author

marcprux commented May 8, 2025

To clarify what the post-install script is doing, following is a result of the output of the command with the default NDK directory:

zap org.swift.swiftpm/swift-sdks % tree swift-6.2-DEVELOPMENT-SNAPSHOT-2025-05-07-a-android-0.1.artifactbundle/swift-android/ndk-sysroot
└── usr
    ├── include -> ~/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include
    └── lib
        ├── aarch64-linux-android -> ~/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android
        ├── arm-linux-androideabi -> ~/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/arm-linux-androideabi
        ├── x86_64-linux-android -> ~/Library/Android/sdk/ndk/27.0.12077973/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/x86_64-linux-android
        ├── swift
        │   └── android
        │       ├── aarch64
        │       │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-aarch64/android/aarch64/swiftrt.o
        │       ├── armv7
        │       │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-armv7/android/armv7/swiftrt.o
        │       └── x86_64
        │           └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-x86_64/android/x86_64/swiftrt.o
        └── swift_static
            └── android
                ├── aarch64
                │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-aarch64/android/aarch64/swiftrt.o
                ├── armv7
                │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-armv7/android/armv7/swiftrt.o
                └── x86_64
                    └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-x86_64/android/x86_64/swiftrt.o

Alternative NDK locations can be specified by overriding the ANDROID_NDK_HOME, like so:

zap org.swift.swiftpm/swift-sdks % ANDROID_NDK_HOME="/opt/homebrew/share/android-ndk" swift-6.2-DEVELOPMENT-SNAPSHOT-2025-05-07-a-android-0.1.artifactbundle/swift-android/scripts/setup-android-sdk.sh
setup-android-sdk.sh: success: ndk-sysroot linked to Android SDK

zap org.swift.swiftpm/swift-sdks % tree swift-6.2-DEVELOPMENT-SNAPSHOT-2025-05-07-a-android-0.1.artifactbundle/swift-android/ndk-sysroot
└── usr
    ├── include -> /opt/homebrew/share/android-ndk/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include
    └── lib
        ├── aarch64-linux-android -> /opt/homebrew/share/android-ndk/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android
        ├── arm-linux-androideabi -> /opt/homebrew/share/android-ndk/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/arm-linux-androideabi
        ├── x86_64-linux-android -> /opt/homebrew/share/android-ndk/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/x86_64-linux-android
        ├── swift
        │   └── android
        │       ├── aarch64
        │       │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-aarch64/android/aarch64/swiftrt.o
        │       ├── armv7
        │       │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-armv7/android/armv7/swiftrt.o
        │       └── x86_64
        │           └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift-x86_64/android/x86_64/swiftrt.o
        └── swift_static
            └── android
                ├── aarch64
                │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-aarch64/android/aarch64/swiftrt.o
                ├── armv7
                │   └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-armv7/android/armv7/swiftrt.o
                └── x86_64
                    └── swiftrt.o -> ../../../../../../swift-resources/usr/lib/swift_static-x86_64/android/x86_64/swiftrt.o

marcprux and others added 2 commits May 17, 2025 20:03
* Add static libraries to post-install script

* Add post-install script to SDK bundle

* Add post-install script to SDK bundle

* Update submodules
@marcprux
Copy link
Author

As of 72964f5, the Android SDK is now being built in a Docker container.

marcprux and others added 3 commits June 12, 2025 21:19
* Run the compiler validation suite for Android

* Add --build-compiler option

* Add --cross-compile-build-swift-tools=0 from swiftlang/swift#38441

* Build with --build-llvm=0

* Check out Yams for Swift 6.1.1 build

* Add --llvm-ninja-targets-for-cross-compile-hosts=help

* Install pre-requisites

* Install build prerequisites

* Only setup local toolchain if build-compiler is not 0

* Fix Yams version checkout

* Quote arguments to build scripts

* Permit empty host-toolchain argument in build.sh

* Fix check for BUILD_COMPILER

* Re-order Docker PATH to system clang is used before Swift toolchain clang

* Use --host-test to skip attempt to test on connected device/emulator

* Install clang in Dockerfile

* Add --skip-test-linux flag to build

* Remove more folders to free up space

* Update patches

* Add docker-specific CI variants that run the compiler validation tests

* meaningless edit

* Update pull_request.yml to stop the 6.1 release builds

* Update pull_request.yml to really only build the full docker compiler with tests

* Update README.md with meaningless edit to bump build

* Update build.sh to skip testing XCTest for linux, as a handful of the linux tests fail for some reason

* Update build.sh to disable building libTesting and for 16K memory pages, as both don't work yet

* Update build.sh to skip testing Foundation for linux, which requires building SwiftPM from source first

* Update build.sh to not build each arch in a separate build directory

* Update Dockerfile to use clang 19 instead

* Add self-hosted runner CI variant

* Fix CI workflow syntax

* Update CI runner config

* Update CI runner config

* Fix name of self-hosted CI runner

* Re-order arch run sequence

* Update CI for self-hosted runner

* Build for self-hosted runner

* Add swift-6.2-branch to self-hosted run matrix

* Update build.sh to disable aarch64 temporarily, as we know it built fine

* Update pull_request.yml to disable non-compiler builds, as they all work, and comment out self-hosted till we know armv7 works

* Run compiler validation on self-hosted with increased timeout

* Update build-docker to only build for armv7

* Update pull_request.yml to disable self-hosted builds temporarily

* Update pull_request.yml to build the full compiler in github runners with the armv7 stdlib and tests

* Update build-docker to build for aarch64 first

* Update toolchain-vars.sh to use older trunk snapahot toolchain that didn't crash

* Update toolchain-vars.sh to only download the latest release compiler if building the Swift compiler from source

* Update build-docker to only build for armv7 again

* Update toolchain-vars.sh to fix setting branch variable

* Update build-docker to build for all three supported arches

* Update pull_request.yml to try self-hosted runs again

* Update build.sh to only install SDK components and remove linux stdlib

* Tolerate missing linux folder when attempting to clean up unnecessary build artifacts

* Fix extraCLIOptions in swift-toolset.json

* gcpdw

* Fix patch application

* Update apply-patches.sh to remove unused changes

* Update swift-android-testing-except-release.patch to add Testing fix

* Update swift-android.patch to remove unneeded patches

* Update build.sh to try and fix Testing and clean up the bundle more

* Fix README.md

---------

Co-authored-by: finagolfin <finagolfin@tuta.io>
…r tags

Also update patches, particularly to disable failing tests from the compiler validation suite
Update patches and build the compiler from the latest commits, not an older tag
@etcwilde
Copy link
Member

Working through this. One thing to note, https://github.com/swiftlang/swift-docker/blob/main/ci_test.py --

swift-docker/ci_test.py

Lines 79 to 83 in c95bc0c

cmd = [
'docker', 'build', '--no-cache=true',
'-f', dockerfile,
'-t', image_name,
docker_dir
runs the docker build commands from the repository root, not from the subdirectory so scripts will need to be able to accommodate a working directory that is different from the directory where the script lives.

@marcprux
Copy link
Author

I think it should be fine in theory to run the Dockerfile from another folder, but it does require some mount points to be set up (see https://github.com/swift-android-sdk/swift-docker/blob/main/swift-ci/sdks/android/build-docker#L63).

I also notice that the static-linux Dockerfile doesn't build out-of-the-box with a generic command like:

$ docker build --no-cache=true -f swift-ci/sdks/static-linux/Dockerfile .

…
--------------------
  86 |     RUN chmod ugo+x /scripts/*
  87 |     
  88 | >>> COPY resources /resources
  89 |     
  90 |     # Create a user
--------------------
ERROR: failed to build: failed to solve: failed to compute cache key: failed to calculate checksum of ref 19bdc85c-49c5-4899-bc25-2a1cffd11dfe::e0kqjvdnitebn8odzbggham6d: "/resources": not found

Is there some different command I should run to test it?

Copy link
Member

@etcwilde etcwilde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Started going through this. I think my main concerns right now are around cloning repo's that we are already cloning with update-checkout and the patches that haven't been merged into the repos yet. Otherwise it's generally looking fine. I still need to look more closely at the build.sh though.

}

# Defaults
if [[ -z "${SWIFT_VERSION}" ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we downloading these separately? update-checkout is already pulling down yams, curl, and libxml2. Do we need to update the versions there for Android? Is this just to keep those versions clean while applying patches to the versions downloaded by this script?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize update-checkout grabbed these too. I was just following the precedent of the static-linux SDK here:

https://github.com/swift-android-sdk/swift-docker/blob/ed09034cd0d61b32cbacd56b26a38fa8b5e823fe/swift-ci/sdks/static-linux/scripts/fetch-source.sh#L183-L194

I could likely re-use the existing yams and libxml2 checkouts, and probably curl too, but we are building and linking BoringSSL (like the static linux SDK does), so we will need to make sure that works with the version of curl that is being grabbed with update-checkout.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the redundant source checkouts in swift-android-sdk#11. Thanks for spotting it! @al45tair, do you know why the static-linux SDK is checking out libxml2 and curl when it could be re-using the repositories already checked out with update-checkout?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I wanted to have direct control over which versions get built into the Static SDK (which also let me pick new enough versions that they'd work with the other dependencies).

BTW, I'd be careful with BoringSSL. OpenSSL would be a safer choice, I think, simply because BoringSSL doesn't really have releases and so unless you collaborate directly with the BoringSSL maintainers (which is what NIO does), you won't get security fixes and so on. I'm actually thinking of switching the Static SDK over to OpenSSL for that reason — either that, or making it always grab the version NIO is currently using, since the NIO folks already watch for updates to BoringSSL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I wish the BoringSSL maintainers would make it slightly more explicit that using it was a bad idea, rather than merely saying they don't recommend it because of API/ABI stability. The thing I worry about is security bugs, not ABI stability.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had assumed/presumed that since the BoringSSL "version" being used was the "fips-20220613" tag, that that FIPS designation held some significance or potential for some security-related certification.

The other reason I stuck with it is that BoringSSL is what Android itself uses by default as of Android 6.0/API 23, even though they don't expose it as part of the NDK. I'm sure OpenSSL would be fine too, but I've read claims that BoringSSL is "optimized for Android".

making it always grab the version NIO is currently using, since the NIO folks already watch for updates to BoringSSL.

I think this would make the most sense. I wonder if we could harmonize on some source of truth for the version, and perhaps update-checkout could start checking out BoringSSL with this tag.

Is FoundationNetwork going to be based on swift-nio/swift-nio-ssl in the future? If so, then the issue will solve itself eventually 😄

Perhaps @Lukasa has insights that would apply here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had assumed/presumed that since the BoringSSL "version" being used was the "fips-20220613" tag, that that FIPS designation held some significance or potential for some security-related certification.

Yeah, that was my assumption too, but that's (sadly) wrong. @Lukasa, WDYT we should do here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could likely re-use the existing yams and libxml2 checkouts, and probably curl too, but we are building and linking BoringSSL (like the static linux SDK does), so we will need to make sure that works with the version of curl that is being grabbed with update-checkout.

Note that the build-script curl build also links against OpenSSL, but simply does so implicitly, so it only works when natively compiling and the system package is installed, then fails when cross-compiling, so we'll definitely need to add one of those SSL libraries to the build-script config.

I wonder if we could harmonize on some source of truth for the version, and perhaps update-checkout could start checking out BoringSSL with this tag.

update-checkout already checks out swift-crypto, which is based on BoringSSL for most platforms and has a CMake config, so maybe we can just reuse that for curl too?

@@ -21,3 +21,172 @@ jobs:
name: docker-logs
path: |
*.log

android-build:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned about running this on every PR against the swift-docker repository.
At the moment, testing on this repo takes a few minutes maximum, while I suspect that this will increase that it pretty substantially. It might be good to either limit running it to changes to the android subdirectory, or we should make this a dedicated job on Swift CI like the static Linux Musl SDK instead.
CC @shahmishal, @justice-adams-apple

Copy link
Author

@marcprux marcprux Jul 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned about running this on every PR against the swift-docker repository.

Yeah, actually, that android-build CI part was intended to be expunged before this PR would be merged. 😊

It is just there so we can exercise updates to the pull request.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, that sounds good to me. Either Justice or I can probably use it as a template for setting up the Jenkins job if we have questions later, so useful to have in history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants